Predictive performance of multi-model ensemble forecasts of COVID-19 across European nations.

Background: Short-term forecasts of infectious disease burden can contribute to situational awareness and aid capacity planning. Based on best practice in other fields and recent insights in infectious disease epidemiology, one can maximise the predictive performance of such forecasts if multiple models are combined into an ensemble. Here, we report on the performance of ensembles in predicting COVID-19 cases and deaths across Europe between 08 March 2021 and 07 March 2022. Methods: We used open-source tools to develop a public European COVID-19 Forecast Hub. We invited groups globally to contribute weekly forecasts for COVID-19 cases and deaths reported by a standardised source for 32 countries over the next 1-4 weeks. Teams submitted forecasts from March 2021 using standardised quantiles of the predictive distribution. Each week we created an ensemble forecast, where each predictive quantile was calculated as the equally-weighted average (initially the mean and then from 26th July the median) of all individual models' predictive quantiles. We measured the performance of each model using the relative Weighted Interval Score (WIS), comparing models' forecast accuracy relative to all other models. We retrospectively explored alternative methods for ensemble forecasts, including weighted averages based on models' past predictive performance. Results: Over 52 weeks, we collected forecasts from 48 unique models. We evaluated 29 models' forecast scores in comparison to the ensemble model. We found a weekly ensemble had a consistently strong performance across countries over time. Across all horizons and locations, the ensemble performed better on relative WIS than 83% of participating models' forecasts of incident cases (with a total N=886 predictions from 23 unique models), and 91% of participating models' forecasts of deaths (N=763 predictions from 20 models). Across a 1-4 week time horizon, ensemble performance declined with longer forecast periods when forecasting cases, but remained stable over 4 weeks for incident death forecasts. In every forecast across 32 countries, the ensemble outperformed most contributing models when forecasting either cases or deaths, frequently outperforming all of its individual component models. Among several choices of ensemble methods we found that the most influential and best choice was to use a median average of models instead of using the mean, regardless of methods of weighting component forecast models. Conclusions: Our results support the use of combining forecasts from individual models into an ensemble in order to improve predictive performance across epidemiological targets and populations during infectious disease epidemics. Our findings further suggest that median ensemble methods yield better predictive performance more than ones based on means. Our findings also highlight that forecast consumers should place more weight on incident death forecasts than incident case forecasts at forecast horizons greater than 2 weeks. Funding: AA, BH, BL, LWa, MMa, PP, SV funded by National Institutes of Health (NIH) Grant 1R01GM109718, NSF BIG DATA Grant IIS-1633028, NSF Grant No.: OAC-1916805, NSF Expeditions in Computing Grant CCF-1918656, CCF-1917819, NSF RAPID CNS-2028004, NSF RAPID OAC-2027541, US Centers for Disease Control and Prevention 75D30119C05935, a grant from Google, University of Virginia Strategic Investment Fund award number SIF160, Defense Threat Reduction Agency (DTRA) under Contract No. HDTRA1-19-D-0007, and respectively Virginia Dept of Health Grant VDH-21-501-0141, VDH-21-501-0143, VDH-21-501-0147, VDH-21-501-0145, VDH-21-501-0146, VDH-21-501-0142, VDH-21-501-0148. AF, AMa, GL funded by SMIGE - Modelli statistici inferenziali per governare l'epidemia, FISR 2020-Covid-19 I Fase, FISR2020IP-00156, Codice Progetto: PRJ-0695. AM, BK, FD, FR, JK, JN, JZ, KN, MG, MR, MS, RB funded by Ministry of Science and Higher Education of Poland with grant 28/WFSN/2021 to the University of Warsaw. BRe, CPe, JLAz funded by Ministerio de Sanidad/ISCIII. BT, PG funded by PERISCOPE European H2020 project, contract number 101016233. CP, DL, EA, MC, SA funded by European Commission - Directorate-General for Communications Networks, Content and Technology through the contract LC-01485746, and Ministerio de Ciencia, Innovacion y Universidades and FEDER, with the project PGC2018-095456-B-I00. DE., MGu funded by Spanish Ministry of Health / REACT-UE (FEDER). DO, GF, IMi, LC funded by Laboratory Directed Research and Development program of Los Alamos National Laboratory (LANL) under project number 20200700ER. DS, ELR, GG, NGR, NW, YW funded by National Institutes of General Medical Sciences (R35GM119582; the content is solely the responsibility of the authors and does not necessarily represent the official views of NIGMS or the National Institutes of Health). FB, FP funded by InPresa, Lombardy Region, Italy. HG, KS funded by European Centre for Disease Prevention and Control. IV funded by Agencia de Qualitat i Avaluacio Sanitaries de Catalunya (AQuAS) through contract 2021-021OE. JDe, SMo, VP funded by Netzwerk Universitatsmedizin (NUM) project egePan (01KX2021). JPB, SH, TH funded by Federal Ministry of Education and Research (BMBF; grant 05M18SIA). KH, MSc, YKh funded by Project SaxoCOV, funded by the German Free State of Saxony. Presentation of data, model results and simulations also funded by the NFDI4Health Task Force COVID-19 (https://www.nfdi4health.de/task-force-covid-19-2) within the framework of a DFG-project (LO-342/17-1). LP, VE funded by Mathematical and Statistical modelling project (MUNI/A/1615/2020), Online platform for real-time monitoring, analysis and management of epidemic situations (MUNI/11/02202001/2020); VE also supported by RECETOX research infrastructure (Ministry of Education, Youth and Sports of the Czech Republic: LM2018121), the CETOCOEN EXCELLENCE (CZ.02.1.01/0.0/0.0/17-043/0009632), RECETOX RI project (CZ.02.1.01/0.0/0.0/16-013/0001761). NIB funded by Health Protection Research Unit (grant code NIHR200908). SAb, SF funded by Wellcome Trust (210758/Z/18/Z).

Collaborative Hubs: Making the Most of Predictive Epidemic Modeling

[...]models can vary in terms of what data they use, what they assume about transmission, and what analytic approach they use to produce projections. Because of this, relying on one model is dangerous because there is no guarantee that one model's choices and assumptions will yield an accurate prediction. In many fields, there is a long tradition of combining multiple models to mitigate this limitation by providing a single prediction that summarizes the view of the participating models.7 There has been a growing interest in using ensemble methodologies in epidemiology, with notable efforts in forecasting, risk prediction, causal inference, and decision-making.8-12 COORDINATION, COLLABORATION, AND EVALUATION A modeling "hub" is a consortium of research groups organized around a particular scientific challenge. The US COVID-19 Forecast Hub ensemble (including many component models) has struggled to produce accurate forecasts of cases and hospitalizations during periods of rapidly changing epidemic dynamics, such as the US peak of the winter wave in early 2021 or the rapid increases associated with the Delta variant in summer 2021 or in winter 2021-2022.3 Likewise, although longer-term projections from the COVID-19 Scenario Modeling Hub projected a Delta-associated resurgence in the United States, the ensemble significantly underestimated its speed and size, even though there were no clear deviations from scenario assumptions.13 However, even when projections are wrong, the hubs play a role in enhancing the scientific rigor and integrity of epidemic modeling. [...]operationally, there is value in developing procedures that harness the insights of a diverse network of scientists while guarding against groupthink and overconfidence.12 As researchers, system developers, and public health officials who have been deeply involved in the real-time operation of modeling hubs duringthe COVID-19 pandemic and prior epidemics, we believe the hub approach is a vital path forward for predictive disease modeling efforts.

Public health considerations for transitioning beyond the acute phase of the COVID-19 pandemic in the EU/EEA.

Many countries, including some within the EU/EEA, are in the process of transitioning from the acute pandemic phase. During this transition, it is crucial that countries' strategies and activities remain guided by clear COVID-19 control objectives, which increasingly will focus on preventing and managing severe outcomes. Therefore, attention must be given to the groups that are particularly vulnerable to severe outcomes of SARS-CoV-2 infection, including individuals in congregate and healthcare settings. In this phase of pandemic management, a strong focus must remain on transitioning testing approaches and systems for targeted surveillance of COVID-19, capitalising on and strengthening existing systems for respiratory virus surveillance. Furthermore, it will be crucial to focus on lessons learned from the pandemic to enhance preparedness and to enact robust systems for the preparedness, detection, rapid investigation and assessment of new and emerging SARS-CoV-2 variants. Filling existing knowledge gaps, including behavioural insights, can help guide the response to future resurgences of SARS-CoV-2 and/or the emergence of other pandemics. Finally, 'vaccine agility' will be needed to respond to changes in people's behaviours, changes in the virus, and changes in population immunity, all the while addressing issues of global health equity.

How to detect and reduce potential sources of biases in studies of SARS-CoV-2 and COVID-19.

In response to the coronavirus disease (COVID-19) pandemic, public health scientists have produced a large and rapidly expanding body of literature that aims to answer critical questions, such as the proportion of the population in a geographic area that has been infected; the transmissibility of the virus and factors associated with high infectiousness or susceptibility to infection; which groups are the most at risk of infection, morbidity and mortality; and the degree to which antibodies confer protection to re-infection. Observational studies are subject to a number of different biases, including confounding, selection bias, and measurement error, that may threaten their validity or influence the interpretation of their results. To assist in the critical evaluation of a vast body of literature and contribute to future study design, we outline and propose solutions to biases that can occur across different categories of observational studies of COVID-19. We consider potential biases that could occur in five categories of studies: (1) cross-sectional seroprevalence, (2) longitudinal seroprotection, (3) risk factor studies to inform interventions, (4) studies to estimate the secondary attack rate, and (5) studies that use secondary attack rates to make inferences about infectiousness and susceptibility.

Estimating internationally imported cases during the early COVID-19 pandemic.

Early in the COVID-19 pandemic, predictions of international outbreaks were largely based on imported cases from Wuhan, China, potentially missing imports from other cities. We provide a method, combining daily COVID-19 prevalence and flight passenger volume, to estimate importations from 18 Chinese cities to 43 international destinations, including 26 in Africa. Global case importations from China in early January came primarily from Wuhan, but the inferred source shifted to other cities in mid-February, especially for importations to African destinations. We estimate that 10.4 (6.2 - 27.1) COVID-19 cases were imported to these African destinations, which exhibited marked variation in their magnitude and main sources of importation. We estimate that 90% of imported cases arrived between 17 January and 7 February, prior to the first case detections. Our results highlight the dynamic role of source locations, which can help focus surveillance and response efforts.

Practical considerations for measuring the effective reproductive number, Rt.

Estimation of the effective reproductive number Rt is important for detecting changes in disease transmission over time. During the Coronavirus Disease 2019 (COVID-19) pandemic, policy makers and public health officials are using Rt to assess the effectiveness of interventions and to inform policy. However, estimation of Rt from available data presents several challenges, with critical implications for the interpretation of the course of the pandemic. The purpose of this document is to summarize these challenges, illustrate them with examples from synthetic data, and, where possible, make recommendations. For near real-time estimation of Rt, we recommend the approach of Cori and colleagues, which uses data from before time t and empirical estimates of the distribution of time between infections. Methods that require data from after time t, such as Wallinga and Teunis, are conceptually and methodologically less suited for near real-time estimation, but may be appropriate for retrospective analyses of how individuals infected at different time points contributed to the spread. We advise caution when using methods derived from the approach of Bettencourt and Ribeiro, as the resulting Rt estimates may be biased if the underlying structural assumptions are not met. Two key challenges common to all approaches are accurate specification of the generation interval and reconstruction of the time series of new infections from observations occurring long after the moment of transmission. Naive approaches for dealing with observation delays, such as subtracting delays sampled from a distribution, can introduce bias. We provide suggestions for how to mitigate this and other technical challenges and highlight open problems in Rt estimation.

Estimating clinical severity of COVID-19 from the transmission dynamics in Wuhan, China.

As of 29 February 2020 there were 79,394 confirmed cases and 2,838 deaths from COVID-19 in mainland China. Of these, 48,557 cases and 2,169 deaths occurred in the epicenter, Wuhan. A key public health priority during the emergence of a novel pathogen is estimating clinical severity, which requires properly adjusting for the case ascertainment rate and the delay between symptoms onset and death. Using public and published information, we estimate that the overall symptomatic case fatality risk (the probability of dying after developing symptoms) of COVID-19 in Wuhan was 1.4% (0.9-2.1%), which is substantially lower than both the corresponding crude or naïve confirmed case fatality risk (2,169/48,557 = 4.5%) and the approximator1 of deaths/deaths + recoveries (2,169/2,169 + 17,572 = 11%) as of 29 February 2020. Compared to those aged 30-59 years, those aged below 30 and above 59 years were 0.6 (0.3-1.1) and 5.1 (4.2-6.1) times more likely to die after developing symptoms. The risk of symptomatic infection increased with age (for example, at ~4% per year among adults aged 30-60 years).

Using observational data to quantify bias of traveller-derived COVID-19 prevalence estimates in Wuhan, China.

BACKGROUND: The incidence of coronavirus disease 2019 (COVID-19) in Wuhan, China, has been estimated using imported case counts of international travellers, generally under the assumptions that all cases of the disease in travellers have been ascertained and that infection prevalence in travellers and residents is the same. However, findings indicate variation among locations in the capacity for detection of imported cases. Singapore has had very strong epidemiological surveillance and contact tracing capacity during previous infectious disease outbreaks and has consistently shown high sensitivity of case-detection during the COVID-19 outbreak. METHODS: We used a Bayesian modelling approach to estimate the relative capacity for detection of imported cases of COVID-19 for 194 locations (excluding China) compared with that for Singapore. We also built a simple mathematical model of the point prevalence of infection in visitors to an epicentre relative to that in residents. FINDINGS: The weighted global ability to detect Wuhan-to-location imported cases of COVID-19 was estimated to be 38% (95% highest posterior density interval [HPDI] 22-64) of Singapore's capacity. This value is equivalent to 2·8 (95% HPDI 1·5-4·4) times the current number of imported and reported cases that could have been detected if all locations had had the same detection capacity as Singapore. Using the second component of the Global Health Security index to stratify likely case-detection capacities, the ability to detect imported cases relative to Singapore was 40% (95% HPDI 22-67) among locations with high surveillance capacity, 37% (18-68) among locations with medium surveillance capacity, and 11% (0-42) among locations with low surveillance capacity. Treating all travellers as if they were residents (rather than accounting for the brief stay of some of these travellers in Wuhan) contributed modestly to underestimation of prevalence. INTERPRETATION: Estimates of case counts in Wuhan based on assumptions of 100% detection in travellers could have been underestimated by several fold. Furthermore, severity estimates will be inflated several fold since they also rely on case count estimates. Finally, our model supports evidence that underdetected cases of COVID-19 have probably spread in most locations around the world, with greatest risk in locations of low detection capacity and high connectivity to the epicentre of the outbreak. FUNDING: US National Institute of General Medical Sciences, and Fellowship Foundation Ramon Areces.

Identifying Locations with Possible Undetected Imported Severe Acute Respiratory Syndrome Coronavirus 2 Cases by Using Importation Predictions.

Cases of severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) infection exported from mainland China could lead to self-sustained outbreaks in other countries. By February 2020, several countries were reporting imported SARS-CoV-2 cases. To contain the virus, early detection of imported SARS-CoV-2 cases is critical. We used air travel volume estimates from Wuhan, China, to international destinations and a generalized linear regression model to identify locations that could have undetected imported cases. Our model can be adjusted to account for exportation of cases from other locations as the virus spreads and more information on importations and transmission becomes available. Early detection and appropriate control measures can reduce the risk for transmission in all locations.